Linear Programming for Finite State Multi-Armed Bandit Problems

1. iBtrodactfMu An important sequential control problem with a tractable solution is the multi-armed bandit problem. It can be stated as follows. There are N independent projects, e.g., statistical populations (see Robbins 19S2), gambling machines (or bandits) etc.. The state of the pth of them at time t is denoted by x,it) and it belongs to a set of possible states S, which in this paper is assumed to be finite. Let 5, =: { 1 , . . . , A,}. At each point in time one can work on one project only and if the i>th of them is selected, one receives a reward r(f) = r'it) and its state changes according to a stationary transition rule: py = P(x,(/+ l)=y|x^(/)= /) whfle the states of all other projects remain unchanged: x,(< -f1) = x^it) iS K¥= P. Let xit) = (x, ( f ) , . . . , x^it)) and let wit) denote the project selected at time /. The states of all projects are observable and the problem is to choose ir(/) as a function of xit), so as to maximize the expected total discounted reward, given an initial state x(0):